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ABSTRACT 

The primary purpose of this study was to examine the 
effects of replacing multiple-choice questions in tutorial software 
practice interactions with a learning strategy (i.e., 
learner-generated summaries) on immediate recall. Subjects were 35 
students— 32 females and 3 males—from an undergraduate computer 
course in the College of Education at Brigham Young University 
(Utah). The intervention for all treatment groups consisted of an 
interactive videodisk in developmental biology which had been 
developed using the traditional approach of instructional segment 
followed by multiple choice questions. Students were randomly 
assigned to three treatment groups and a pretest was administered. 
Students in Group 1 then answered multiple-choice questions after 
each of four instructional modules. In Group 2, students were asked 
to generate a written summary, which they typed in their own words on 
the computer, about the information presented in the module. Students 
m Group 3 were also asked to write a summary of the material just 
presented to them; however, after they wrote their summaries, they 
were provided feedback in the form of a system-generated summary of 
all of the main points in the module and told to compare the 
system-generated summary with their own. A posttest was administered 
and students were interviewed to determine whether they had used 
self-initiated learning strategies in addition to those designed in 
the interventions. It was found that replacing multiple-choice 
questions with learner-generated summaries not only did not 
facilitate immediate recall of the information presented, but reduced 
the efficiency of the learning process, since students who generated 
their own summaries took significantly longer to complete the 
instruction. (33 references) (BBM) 
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Hie Efifects of Integrating a Leaniing Strategy in CBI 

Tutorial courseware allegedly enacts a dialog between the program designer 
and the learner. According to Soulier (1988), "dialog frames" present information 
to the learner, as well as cany out an interactive dialog/feedback between the 
garner and the computer. Soulier borrows the terminology of "dialog frames" 
from Alfred Bork (1980), who defined dialog as "a 'conversation' between a student 
and a teacher, where the student is at the computer display and the teacher is 
conducing the dialog through the medium of a computer program" (p. 15) 

Soulier (1988) explains that dialog frames in computer-based instruction (CBI) 
usually have three parts: rules, examples and interactions. The information or 
subject matter IS usually presented in the first two parts. The focus of this study is 
concerned with the third part, the practice component, or as Soulier refers to it, the 
interaction function. 

According to Alessi and Trollip (1985), the most common method of CBI 
mte/action is for the computer to pose questions the learner must answer. 
Quesbons serve a number of purposes. They keep the student attentive to the lesson, 
force the student to practice and assess the student's recall and comprehension. In 
addition student responses to the questions determine the sequencing for the 
remainder of the lesson. 6 t • w « 

Questions can be categorized into t,^o basic types: selected-response and 
constructed-response. Selected-response questions are those for which the student 
chooses the correct response or responses from a list of two or more options. These 
include trueyfalse, matching, multiple-choice, and marking. Multiple-choice 
Items are the most common type of seiected-response questions used in CBI (Alessi 
«s lYollip, 1985). Constructed-response questions are open-ended questions that 
require the student to produce rather than select a response. The three major types 
of constnwted-reaponae questions are completion, short-answer, and essay. In 
bBi, the most common are completion and short-answer questions 

T * * microcomputer courseware, 

Jonassen (1988) states that one of the prt)blem8 with most tutorial courseware is that 
the level of mental processing required of learners is too shallow. He claims this 
wetness IS due to the multiple-choice and short-answer questions predominantly 
used in CBI interactions. He states that the nature of the interactivity is usually 
recognition and recall of information presented on the previous screen, with Uttle, 
If any. attempt to relate the information to prior knowledge, so that it will be 
meaningful and therefore retrievable by the learner. Jonassen (19«8) states 
Tutorial courseware is basically a mis-application of the frogrammed 
learning model of instructional design, which has been the dominant 
paradigm in the field of educational technology for nearly three decades. Since 
the programmed learning model is easily confused with the procedure or 
technique of programmed instruction, it is better conceptually defined by the 
mathemagenic hypothesis. »- ^ "J 

Mathemagenic behaviors are 'those student activities that are relevant to 

lwL'^!rr objectives in specified situations or 

places, that 18, those which "give birtJi to learning" (Rothkopf cited in 

fr «<^«o/ding to the hypothesis, can be controlled 
or manipulated by specific design attributes of instruction. The form or 
structure of instruction or the activities stimulated by it induce the necessarjr 
cognitive operations to produce tiie desired learning. ... The purpose of 

lo«r„?„T^";' ««"^"«f' a« i"«erted questions (tiie basis of programmed 
learning), is to control the way in which information is ti-ansformed and 
encod. i into memory. It is therefore a reductive approach to learning, which 
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regards learners as active performers whose mental behavior should be strictly 
controlled by the activities imposed by the lesson, (p. 152) 
Jonassen continues by stating that one of the problems with computer 
applications of the mathemagenic model is that the level of processing normally 
produced by mathemagenic behaviors (especially programmed learning) is too low 
or shallow. 

To solve the problem, Jonassen (1988) recommends tutorial courseware be 
developed based on a more constructive conception of learning, that is, instruction 
which focuses more on cognition, requiring deeper levels of mental processing. 

In order to increase the level of mental processing, Jonassen (1988) 
recommends that tutorial courseware should enhance meaning of course 
information by stimulating the learners to call up and apply what they already 
know. He suggests that this deeper processing can be achieved by incorporating 
learning strategies into the courseware. These learning strategies sometimes 
referred to as cognitive strategies, are mental operations or procedures that 
represent a wide range of cognitive activities including underlining main ideas, 
generating examples or analogies, summarizing, outlining, etc., which the 
student may use to acquire, retain, and retrieve different kinds of knowledge 
(Rigney, 1978). They are designed to increase the number of "links" between 
presented information and existing knowledge in order to enhance retention and 
to allow the student to process information generatively (Wittrock, 1974). They are 
always performed by the leamer-at the initiation of the student or the instructional 
system. 

According to Wfiinstein and Mayer (1986), in recent years increased attention 
has been placed on the role of the learner as an active participant in the teaching- 
leammg act. Many articles suggest that the effects of instruction depend partly on 
pnor knowledge, and the learner's active cognitive processing during the 
learning process (Anderson, Spiro & Montague, 1977; Cook & Mayer, 1983; 
Dansereau, 1985; Jones, Amiran & Katims, 1985; Mayer, 1984; Ryan, 1981; 
Weinstein, 1978; Weinstein & Underwood, 1985; Wittrock, 1974, 1978). Wittrock's 
(1974) generative learning hypothesis assumes that when faced with instructional 
stimuli, learners construct and assign meaning to the stimuli on the basis of prior 
learning. The meaning generated by learners for information they receive is 
individual and cannot be controlled by the author. According to Wittrock (cited in 
Jonassen, 1988), learning is not "a passive reception of someone else's 
organizations and abstractions" (p. 153). Instead, learning is an active, 
constructive process. Jonassen (1985) states, 

The purpose of generative strategies is to provide learners with active, 
instructive skills for proacUvely transferring prior knowledge. Meaning 
(knowledge, if you prefer) is learner-constructed, not media-controlled. These 
[learning] strategies, such as paraphrasing, generating questions, and 
imaging, ore all individual processes for constructing meaning, 
(p. 31) 

Jonassen (1986) explains that microcomputers are especially amenable to the 
inclusion of learning strategies because they can accept, store, and manipulate a 
variety of mput, and they can insist on a response before allowing the learner to 
proceed; something which traditional print media cannot do. For instance, after 
presenting the information, the computer system could direct the learner to go 
through a particular learning strategy, with the computer functioning as an 
electronic notebook." The learner can key in a response which the computer 
evaluates for quantity or existence of key concepts and sometimes manipulates 
through rearranging or mapping and then stores for use by learners as a review or 
retrieval strategy. Jonassen (1988) believes that assessing user input will ensure 
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higher levels of processing than the multi-option recognition or recall tasks 
included in most tutorial courseware. 

Jonassen (1988) recommends that "the simplest method for integrating 
[learning] strategies in courseware is to replace the adjunct, mathemagenic 
activities that are normally included as practice in courseware with specific 
mformation processing or perhaps metaleaming strategies" (p. 160). More 
specifically he suggests: 

For example, rather than inserting multiple-choice questions to test immediate 
recall or comprehension of information in a program, you might periodically 
insert any of the following directions: Summarize in your own words the ideas 
presented; recall and record key ideas and use them to create analogies, 
outhnes, or cognitive maps; draw a picture or generate a mental image of the 
(p 160)™**^*^*'' °' implications of the material that you are studying. 

In discussing the rationale for replacing multiple-choice and short-answer 
questions in CBI interaction frames, Jonassen (1988) explains that "only deeper, 
semantic processing of information requires the learner to access prior knowledge 
in order to interpret new material" (p. 153). makes the point that integrating 
cognitive strategies into CBI interactions will facilitate higher level learning 
outcomes, such as comprehension and understanding, and will also increase the 
amount of information that is recalled. Because learners assign more meaning to 
the information presented during the tutorial, they will be able to remember more 
Jonassen continues. 

It is exactly this level of meaningful learning that is most fi-equently missing 
from tutonal types of courseware. The emphasis is on practice of associations 
in working memory based only on information recently presented in the 
courseware. In interacting with the tutorial courseware, learners are too 
seldom required to access prior knowledge in order to interpret the information 
that IS presented, (p. 153) 

The primary purpose of this study was to examine the effects of replacing 
multiple-choice questions in tutorial courseware practice interactions with a 
learning strategy (learner-generated summaries) on immediate recall. The 
study was designed to investigate empirically the approach recommended by 
Jonassen (1988) to integrate learning strategies by replacing the traditional 
multiple-choice questions in CBI interactions. 

Although several studies within the last few years have investigated the effects 
of integrating individi il learning strategies in instructional systems and 
developing mstructionai systems that train learners how to use a particular 
A^^f i'^^^F ^einstain & Mayer, 1985; Dansereau et al., 1979; O'Neil, A978- 
O-Neil & Spielberger, 1979), the practice of integrating learning strategies in CBI 
interactions has not received much attention in the research literature. The few 
studies which have been conducted had a different focus than the present study For 
ei^mple, tnese studies either dealt with a different learning outcome than 
immediate recall or the study tried to validate a different type of learning strategy 
(for example, see Eucker, 1984; Allen, 1982; Wilshire, 1990). The findings from 
these studies did not support the hypothesis that integrating a learning strategy in 
CBI would be more effective in facilitating learning than the traditional approach 
used by the comparison group(s). 

Several studies have examined different aspects of student-generated 
summaries (Annis, 1986; Brown, Campione, & Day, 1981; Day, cited in Brown. 
Campione^ & Day. 1981; Ballesteros. 1986); however, none have examined the 
specific effects of integrating this strategy in CBI interactions on immediate 

The following hypotheses and research questions were addressed in this study: 
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Hypotheses 

1. Learner-generated summaries in tutorial courseware will be more effective in 
facilitating learners' ability to immediately recall information than multiple- 
choice interactions. 

2. A system-generated summaiy provided to learners (as a form of feedback) 
after they generate their own summaries will be more effective in facilitating 
learners' ability to immediately recall information than not providing a 
system-generated summary. 

Research Questions 

1. Will students who are required to generate summaries take significantly more 
time to complete the intervention than students who respond to multiple-choice 
interactions? 

2. Will students who are provided a system-generated summary type fewer words 
in their summaries than students who are not provided a system-generated 
summary? 

3. Will typing rather than writing the learner-generated summaries affect the 
quantity or quality of the summaries? 

4. To what degree will students use self-initiated learning strategies during the 
intervention? 



Method 

Subjects 

Subjects were obtained from an undergraduate computer course (Computers in 
Education) offered in the College of Education at Brigham Young University in 
Prove, Utah. They were required to complete the intervention for this study as one 
of their assignments in the course. Thirty-fivs students participated in the study 
thirty-two females and three males. 

Materials 

The intervention for all treatment groups consisted of an interactive videodisc 
tutorial on the subject matter of developmental biology, developed by WICAT 
Incorporated in 1979, under a grant from the National Science Foundation 
(Bunderson, Olsen & Baillio, 1981). The hardware consisted of a Macintosh-based 
interactive videodisc system. 

The instruction, which was developed for university-level students, dealt 
primarily with verbal information learning outcomes. This explanative material 
(Mayer, 1989) consisted of an introduction to basic developmental biology concepts 
such as DNA, RNA. genes, chromosomes, etc. It was selected for several reasons 
f^rst, the researcher believed that most students had very little prior knowledge 
about the content, which was desirable in determining whether a learning effect 
occurred as a result of the treatment Second, the instruction on the videodisc had 
been developed using the traditional type of CBI interactions: an instructional 
segment followed by multiple-choice questions. Finally, since the content area 
was not related to the course in which students were enrolled, the researcher 
beheved the students would not make extensive use of self-initiated learning 
strategies as they completed the instruction. 

The interaction frames and the courseware that ^iontrolled the videodisc were 
developed on the computer by use of the authoring program The Best of Course of 
Action, developed by Authorware, Incorporated (Auth<)rware, 1987) 
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Research Design 

The research was conducted using a Pretest-Posttest Comparison Group 
Design. The students were randomly assigned to one of three treatment groups 
The mstruction delivered during the intervention was the same for all three 
treatment groups-four modules consisting of videodisc motion sequences and 
several still frames concerning developmental biology. Students in Group 1 
answered multiple-choice questions after each instructional module. In Group 2 
students were asked to generate a written summary, which they typed in their own 
words on the computer, about the information presented in each module. Students 
m Group 3 were also asked to type a written summaiy of the information just 
presented to them; however, after they wrote their summaiy, they were provided 
feedback m the form of a system-generated summary which summarized all of the 
main pomts in the module and were told to compare the system-generated 
summary with their own to determine if they had left out any important ideas 
nr.» . ! *"°®P®"*^«"^ variable was the type of learner response required during the 
CBI interaction following the instructional modules. The dependent variable for 
the hypotheses was the posttest score. 

Instruments 

A pretest was developed to assess the preexisting knowledge of students on the 
content of developmental biology in order to determine if the three treatment groups 
were eqmvalent as a result of random assignment to groups. A posttest was 
developed using items selected from a posttest developed by WICAT Inc., based on 
the information presented in the videodisc. The test consisted of multiple-choice 
matching, true/false, and short-answer test items which assessed recall leamine 
outcomes. * 

Internal consistency reliability measures were computed on the test scores to 
assess the reliability of the results. The reliability (Cronbach Alpha) coefficients 
for the pretest and posttest scores were .76 and .72, respectively. 

Procedure 

The researcher administered the intervention for each student individually 
dunng a one-hour time slot. When the student arrived, the researcher briefly 
explamed the nature of the study and then requested him or her to sign a 
participaUon consent form and complete the pretest. The researcher then gave the 
student a brief overview on how to use the instructional system. 

After the orientation, the students were presented a module of instruction 
consisting of several stiW frames and a motion sequence. The students were then 
asked to either answer multiple-choice questions or generate a summary of the 
mformation just presented. Students in Group 1 who selected the correct answer 
were given a short statement affirming the correct selection. Students who selected 
the wrong answer were told their answer was incorrect and then given the correct 
answer. Students m Group 2 were asked to generate a summaiy of the material just 
presented. After the student typed his or her summaiy, the instructional system 
presented the next module. Students in Group 3 were given the same instiictions as 
those m Group 2; however, immediately after the students typed their summary, the 
system presented a complete summary of all the main ideas presented as a form of 
teeaback. Students were asked to compare their summaries to the system- 
generated summaiy to determine if they had left out any important information or 
concepts. This pattern continued for the remaining three modules. Students in 
Group 1 answered a total of 20 multiple-choice practice items, and students in 
Groups 2 and 3 generated four summaries during the entire intervention. 

After the students completed the intervention, they took the posttest Stiidents 
took an average of 60 minutes to complete the entire process. 
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Data Analysis 

The hypotheses were analyssed using a planned, orthogonal comparison 
statistical design. Information related io the first two research questions (minutes 
of elapsed time and average number of words written in the learner-generated 
summaries groups) was analyzed for the three treatment groups using one-way 
analysis of variance. The last two research questions were analyzed using 
descriptive statistics. 

Results 

Of the 35 subjects, 12 were in Group 1, which answered multiple-choice 
questions, 11 were in Group 2 which generated summaries without feedback, and 12 
were in Group 3 which generated summaries and received feedback. Means, 
standard deviations, and ranges were computed on (a) the number of correct 
answers on the pretest (20 points possible), (b) number of correct answers on the 
posttest (33 points possible), (c) instructional time, and (d) number of words in 
student-generated summaries (for Groups 2 and 3 only). Table 1 summarizes these 
computations. 

Table 1 

Group Means, Standard Deviations, and Ranges for Pretest Scores, Posttest Scores, 
Instructional Time, and Number of Words in Student-Generated Summaries 

Numbsr of 

Treatment Pretest Posttest Instructional Words in 

Group Scores Scores Time (minutes) Summaries 



Group 1 (Multiple-choice items) 



Mean 


8.3 


24.3 


25.0 


NA 


SD 


3.1 


2.5 


5.1 


NA 


Range 


2-12 


18-28 


19-33 


NA 



Group 2 (Student-generated sur/smaries without feedback) 



Mean 


8.2 


23.2 


35.6 


205.U 


SD 


4.6 


3.7 


12.0 


74.2 


Range 


1-14 


17-28 


2064 


87^ 



Group 3 (Student-generated summaries with feedback) 



Mean 


8.9 


24.0 


34.9 


236.2 


SD 


4.2 


4.1 


8.5 


116.6 


Range 


2-15 


16^0 


17-53 


77-504 
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One-way analysis of variance revealed that the differences between means on 
the pretest scores were not statistically significant, indicating that the three groups 
were equivalent in preexisting knowledge about the material presented in the 
instruction, F (2, 32) s .11, p s .896. 

The first hypothesis (learner-generated summaries groups will score hi^er 
than the multiple-choice group) was tested using a complex, orthogonal, planned 
comparison analysis (Keppel & Zedeck, 1989), in which the average of the means 
for the two learner-generated summaries groups (Groups 2 & 3) was compared with 
the mean of the multiple-choice practice group (Group 1). Results from the analysis 
indicated no significant treatment effects, T (32) = .53, p = .60. 

The second hypothesis (Group 3 will score higher than Group 2) was tested using 
a pairwise, orthogonal, planned comparison analysis, in which the mean of the 
learner-generated summaries without feedback group (Group 2) was compared 
with the mean of the learner-generated summaries with feedback group (Group 3). 
Results from the analysis indicated no significant differences between the two 
groups, T (32) = . 57, p = . 58. 

With regard to the question of instructional time. Table 1 shows that students in 
Group 1 spent an average of 10 minutes less than students in Groups 2 and 3 during 
the instruction. One-way analysis of variance resulted in a significant difference 
between the means of the ti-eatment groups, F (2,32) = 5.3, p = .01. Results of a 
Tukey-HSD post hoc comparison analysis (p < .05) indicated that Groups 2 and 3 
differed significantly from Group 1. 

The second research question sought to determine if there was a significant 
difference in the number of words generated by the students in Group 2 and Group 3, 
based on the concern that students in Group 3 might minimize the words they wrote 
in their self-generated summaries as a result of knowing they would be presented a 
system-generated summary after typing in their summary. Table 1 shows that 
Group 3 generated an average of 30 more words than students in Group 2. A one- 
way analysis of variance showed that the difference between means was not 
statistically significant, F (1, 21) = .572, p = .458. 

The third research question examined whether typing rather than writing 
affected the quality of the students' summaries. To answer this question, the 
researcher asked students who generated their own summaries if they felt typing 
versus writing their summaries affected the quality and quantity of those 
summaries. Of the 23 students who responded, three students indicated that they 
would have written more and their summaries might have been of a higher quality 
had they been allowed to write the summaries on a piece of paper. All other students 
mdicated that typing was not a factor in generating their summaries. Twenty 
students said they would prefer to type their summaries if given a choice, whereas 
three students indicated they would prefer to write their summaries. 

The fourth research question was critical to the final study: If students used 
extensive self-initiated learning strategies in addition to those designed in the 
interventions, the results would perhaps be influenced more by learner variance 
than by treatment variance. The students were interviewed and asked what self- 
initiated learning strategies, if any, tiiey used as they progressed through the 
mstruction. The results as summarized in Table 2, show that 66 percent of the 
students used rehearsal strategies which consist of repeating more than once in 
their minds the material presented to them. However, m Table 2 shows, few 
students used any learning strategies other than the rehearsal technique 
strategies. 
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Response 
Categories 



Fehearsal 



Organizational Strategy 



Mnemonics 



Mental Imagery 
Note Taking 



Number of 
Respondents 



23 
8 



Percentage of 
Respondents 



66% 
23% 
20% 
9% 
6% 



Note: These percentages do not add up to 100 percent because students could 
give more than one response. 



Discussion 

The results of this study did not support the first hypothesis: replacing multiple- 
choice questions with learner-generated summaries in tutorial courseware did not 
result in learners recalling significantly more of the information presented 
during the instruction. In addition, the results show that students in the learner- 
generated summary groups took significantly more time to complete the 
intervention than students in the multiple-choice questions group. 

It was also hypothesized that providing a system-generated summary to 
learners after they generated their own summary would be more effective in 
facilitating learners' ability to immediately recall information than not 
providing a system-generated summary. The results did not support this 
hypothesis. 

What are some of the theoretical implications of these findings? First, becauBe 
of the small number of subjects in each group, learner variance could have had a 
significant influence on the results. The researcher plans to continue this line of 
research with a larger sample size. Second, for the particular recall learning 
outcome assessed in this study, students may need ♦he practice items of multiple- 
choice questions to provide the necessary repetition required for immediate recall 
(Gagne, 1985; Gagne, Briggs & Wager, 1988). Third, the multiple-choice response 
items may have highlighted the key knowledge elements presented in the 
instruction which focused the learners' attention on the information that was 
really important. Fourth, another theoretical issue which may have had some 
influence on the results of this study is the degree of instructional alignment 
between the learning strategy and the learning outcomes (Cohen, 1987). Perhaps 
using learner-generated summaries was not the most appropriate strategy to use 
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for the particular learning outcome measured. Learner-generated summaries 
may have facilitated other higher-level learning outcomes that were not measured 
in this study, such as understanding, transfer, synthesis, etc. Finally, another 
possible explanation which may have influenced the results of this atady was the 
degree to which students wrote effective summaries. Perhaps the subjects in this 
study were much more familiar with responding to short-answer and multiple- 
choice items and hadn't received much practice in writing summaries. 

The results of this study show that replacing multiple-choice questions in 
tutorial courseware with learner-generated summaries does not facilitate the 
specific learning outcome of immediate recall for the specific material used in this 
study. In fact, replacing multiple-choice questions with learner-generated 
summaries in CBI interactions reduced the efficiency of the learning process, 
since students who generated their own summaries took significantly longer to 
complete the instruction. 
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